Clusters are groups of computers that use groups of redundant computing resources in order to provide continued service when individual system components fail. More specifically, clusters eliminate single points of failure by providing multiple servers, multiple network connections, redundant data storage, etc. Clustering systems are often combined with storage management products that provide additional useful features, such as journaling file systems, volume management, multi-path input/output (I/O) functionality, etc. For example, some storage management products such as Veritas Volume Manager support multipathed storage devices, in which a virtual disk device is made available to initiators of I/O, wherein multiple physical paths exist between the virtual disk and the underlying physical storage.
Problems can arise in a cluster from the failure of interconnection components between nodes. A condition called split brain occurs when independent nodes in a cluster become communicatively disconnected from each other, and each falsely assumes that the other is no longer running. The resulting condition can be described as a fence existing between the nodes, wherein there is no communication through the fence. As a result, the node on each side of the fence assumes it has exclusive access to resources including shared storage. To solve the problem of split brain, the node(s) on one side of the fence or the other should be taken offline, or at least denied access to the shared storage. A technique known as I/O fencing can be used to prevent uncoordinated access to the shared storage, and thus mitigate the risks associated with split brain. I/O fencing allows write access for the node(s) on one side of the fence (known as the active cluster) and blocks access to the shared storage from the node(s) on the other side of the fence (that is, the nodes that are not members of the active cluster).
SCSI-3 Persistent Reservations (SCSI-3 PR) is a feature of SCSI-3. (SCSI-3 is a version of the SCSI (Small Computer System Interface) standard for physically connecting and transferring data between computers and peripheral devices, such as hard disks and tape drives.) SCSI-3 PR supports providing multiple nodes with access to a given storage device while simultaneously blocking access for other nodes, and is thus useful in the context of I/O fencing in a clustered environment utilizing shared storage. SCSI-3 PR uses a concept of registration and reservation. Computer systems register with a given SCSI device, and only registered systems can hold a reservation to issue commands to the particular device. Only one reservation can exist amidst multiple registrations. Under SCSI-3 PR, a computer can be blocked from accessing a storage device by removing its registration. SCSI-3 PR reservations are persistent across SCSI bus resets or computer system reboots. In the case of shared storage, a shared storage device can be comprised of multiple underlying SCSI devices, which logical volume management functionality in the clustering and storage management system virtualizes to appear to computer systems as a single storage device. This allows configuration under SCSI-3 PR such that only registered systems can write to the shared storage device.
Virtualization of computing devices can be employed in clustering and in other contexts. One or more virtual machines (VMs or guests) can be instantiated at a software level on physical computers (host computers or hosts), such that each VM runs its own operating system instance. Just as software applications, including server applications such as databases, enterprise management solutions and e-commerce websites, can be run on physical computers, so too can these applications be run on VMs, which can function as servers in a cluster. Some environments support the ability to move guests (VMs) from a first host computer to a second host computer, often with no downtime for the guest operating system and applications. In other words, a running (“live”) VM can be migrated between nodes in the cluster.
A running VM can be migrated from a source host computer to a target host computer in a clustering environment that utilizes I/O fencing under SCSI-3 PR. When this occurs, the running VM and its components and applications are being moved between host computers, each of which has a separate registration with the shared storage. Prior to the migration, the VM runs on the source computer system, and any attempts the VM makes to access the shared storage are made from the source computer system. Thus, the source computer system is the initiator of the attempts to access the shared storage. After the migration, the VM runs on the target computer system, which from this point on is the initiator of any attempts the VM makes to access the shared storage. In other words, the migration of the VM causes an initiator change, which in turn causes a reservation conflict. In practice, the reservations conflict causes I/O attempts (or other SCSI commands) made by the target computer system to the shared storage to fail, causing applications running on the migrated VM to fault. This results in application downtime.
It would be desirable to address this issue.